Assessing Performance in an Innovative Cybersecurity Pilot Course

In 2014, 25% of all organizations polled across industry said the lack of infosec skills were a problem. In 2015, an Enterprise Strategy Group (ESG) survey found that 28% reported a shortage of infosec skills (Trendmicro, 2015). With the growing threat of cybercrime and national security issues, growing the number of qualified cybersecurity professionals has become a national imperative. As the cybersecurity universe is shaped by new technologies, unknown threats, and increasing vulnerability in a dynamic environment, there is an established need to rapidly establish innovative, effective, efficient and responsive cybersecurity education initiatives (Dark & Mirkovic, 2015). One such initiative recently piloted by the Department of Defense is the Cyber Operations Academy Course (COAC). The first pilot began in May 2015 at the Fort McNair campus in Washington D.C. As a six-month immersive course, participants consisted of 20 mostly military personnel from all four branches of the military services, various backgrounds and little if any cyber experience. Employing an authentic problem-based course using cooperative and collaborative learning models, the pilot consisted of instruction in foundations, defensive/offensive operations, programming, social engineering, and skills integration. Leveraging cyber ranges and capture the flag (CTF) activities, the course was also supported by four “fireteam” leads as facilitators, coaches, and subject matter experts. At the end of the course, students developed cyber capabilities and tools, developed and deployed exploits, detected and responded to incidents, and used social engineering to exploit “targets.” In comparison with existing cyber protection teams deployed in DoD installations, the students were as capable and in some cases more capable in comparisons of performance. In pre/post comparisons, students exhibited potentially large knowledge gains. This paper discusses the nature of the course’s pedagogy; the challenge of developing representations of learning outcomes and performance; and the challenges in developing performance-based assessments to authentically and objectively assess students’ knowledge and skills in the context of the course